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ABSTRACT 

To generate a useful means of analyzing think-aloud 
protocols ol individual students, and to determine the extent to 
which practical information might be derived from transcripts and 
videotapes, this project focused on the problem-solving behaviors of 
17 sixth-grade students videotaped while solving routine textbook 
word problems. Using a checklist derived from prior research, 
problem-solving strategies were grouped in the following categories: 
orientation; organization; execution; and verification. Incidents of 
strategy use were calculated as percentages across individuals in the 
sample, and each category was submitted to a separate analysis of 
variance. Analysis allows determination of the effect that strategy 
use within a category had on strategy use across individuals. Within 
a category, particular strategies proved more natural or accessible 
for all students. For example, use of vocalizing calculations was the 
most common of the execution strategies. An opportunity for 
self-correction seemed to play a significant role in determining 
problem-solving behaviors. Overall, application of the strategies 
checklist proved effective for analyzing the protocols, and actions 
were effective for classifying behaviors. Checklist data were 
suitable for analysis, and the analysis provided useful information. 
Two figures and three tables provide study data, (Author/SLD) 
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ABSTRACT 

This project focused on the problem-solving behaviors of sixth grade students who were 
videotaped while solving routine, textbook word problems. The goals of the study were 
to generate a useful means of analyzing the think-aloud protocols of individual students 
and to determine the extent to which practical information might be gleaned from 
transcripts and videotapes concerning problem solving behaviors exhibited across 
individuals. Using a checklist of behaviors adapted from the work of Garofalo, Lester 
and their colleagues, problem-solving strategies were grouped under the categories of 
Orientation, Organization, Execution and Verification. Incidents of strategy-use were 
calculated as percentages across individuals in the sample, and each category of strategies 
was submitted to a separate analysis of covariance. Results of the analyses allowed a 
determination of strategy-use across individuals. Overall, application of the strategies 
checklist to the problem of analyzing protocols and actions proved to be sufficient for 
classifying the problem solving behaviors of individual students. Data from the 
checklists were suitable for statistical analysis, and results of the analysis provided useful 
infonnation concerning strategy-use across individuals. 
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AN APPROACH TO ANALYZING THE VIDEOTAPED PROBLEM SOLVING 
BEHAVIORS OF INDIVIDUAL STUDENTS 

OBJECTIVE 

The National Council of Teachers of Mathematics (1991) has emphasized the 
need for methods of assessment which accurately reflect student problem-solving 
processes. Such assessments should allow teachers to determine the presence or absence 
of student behaviors such as the ability to formulate problems, to solve problems using 
different strategies, and to verify and interpret results. Additionally, teachers are to use 
these assessments as a basis for their instruction. 

This project focused on the problem-solving behaviors of sixth grade students 
who were videotiiped while solving routine, textbook word problems. The goals of the 
study were to generate a useful means of analyzing the think-aloud protocols of 
individual students, and to determine the extent to which practical information might be 
gleaned from transcripts and videotapes concerning problem solving behaviors exhibited 
across individuals. 

THEORETICAL FRAMEWORK 

Various researchers have posited methods of categorizing mathematical problem 
solving behaviors, concentrating primarily on cognitive and itietacognitive factors 
(Schoenfeld, 1985; Lawson & Rice, 1987; Artzt, 1990). Perhaps the most extensive work 
done in this area has been carried out by Garofalo, Lxister and their colleagues (Lester & 
Garofalo, 1982; Garofalo & Lester, 1985; Lester, Garofalo & Kroll, 1989). Garofalo and 
Lester (1985) identified and defined behaviors for four categories (if strategies in 
mathematical problem solving: Orientation, Organization, Execution and Verification. 
Orientation strategies were those that promoted assessment and understanding of the 
problem situation; Organization strategies reflected the planning behaviors of the problem 
solver; Execution strategies were control or regulatory processes used during problem 
solving compulations; finally. Verification strategics provided the problem solver with a 



4 

o 

ERIC 



Analyzing Problem Solving Behaviors 

4 

means of evaluating decisions and outcomes. A later report (Lester, Garofalo & Kroll, 
1989) amplified and further defined these categories by studying the activities of seventh 
grade students solving non-routine word problems. 



Insert Figure I About Here 



The present study employed a checklist of problem-solving strategies (Figure 1), 
which represents a compilation of the latter two sources. The behaviors identified in 
these sources were considered appropriate for this study since the authors had focused 
their analyses on similar-age students and like mathematical tasks. Only the strategy of 
making trade-off decisions (e.g., accuracy vs. time) was left off of the list, since students 
in the present study had no time restrictions placed on their problem solving. 

DATA SOURCE 

Sexton (1987) videotaped 20 sixth grade students independently solving seven 
routine textbook word problems. Students were chosen from a stratified sample v/ithin 
one North Carolina school system. All students were above-average, as indicated by their 
most recent California Achievement Test scores and their previous year's mathematics 
grade; all subjects were volunteers; and all were paid a small stipend for their 
p irticipation. Sexton employed the method of think-aloud protocols in recording the 
problem solving activities of the students. Data were collected for each subject on audio- 
and videotape during 45 minute problem-solving sessions. 

Students were instructed to verbali/.e evei*y thought they had while solving the 
problems. They were also told (hat the rosciircher would not he able to anr 'vcr any 
questions, and that the only comments that would be made to students would be "Tell me 
what you're thinking" if they had been silent for more than five seconds, and "That's 
correct" or "That's incorivct" when the students said they were done working a problem. 
This last point is important for the present study because the presence of performance 
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feedback provided an opportunity to determine which students had solved a problem 
correctly on the first try, and which students had required more than one try to solve a 
problem. Students were allowed to solve two practice problems before solving the 
problems analyzed in the study. No post-session interviews were conducted with the 
students other than having students order the seven problems according to perceived 
difficulty level. 

METHOD 

The present study re-analyzed the videotapes and transcripts of 17 successful 
problem-solvers working on three moderately difficult problems (difficulty level was 
determined by the average time required to solve a problem and the percentage of 
students able to solve it). Each problem included some unique aspect of routine 
mathematical word problems: the presence of extraneous information (Problem 1), the 
ncc'l for quantity conversion (Problem 2), or the utility of using a diagram (Problem 3). 
The following problems were solved: 

Problem 1 Ears of com sell for 5 for $0.95 at the vegetable stand. Mrs. Jones 
bought 10 ears of corn. How much docs each ear of com cost? 

Problem 2 What do 30 oranges cost at 60 cents per do/en? 

Problem 3 In Fred's room, there are 2 shelves on one wall. The first is 3 ft. off 
the floor, the second is 3 fl. above that. This last shelf is 2 ft. from 
the ceiling. How high is the room? 
For each problem, strategy-use for each student was classified as either present or absent 
on the Figure 1 checklist. A limitation of this study is that prohlem-.solving behaviors for 
the sample were only classified by one rater, and therefore no estimate of the reliabilily of 
this classification is possible. 

The sample was divided inlo two groups for each problem: those siuiionis who 
solved a particular problem on the firsi try, and those students who required nuire than 
one try to solve a problem. An inialysis of covariance (ANCOVA) was condueUHl toi 
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each category of strategics (Orientation, Organization, Execution, Verification) across all 
17 studenLs. The independent variables were the number of tries needed to solve a 
problem (one or more than one) and the su*ategies for the different categories (which 
varied for each category). The covariale was the problem that was solved, entered as a set 
of dummy variables due to the categorical nature of the variable. The dependent variable 
was the percentage of students in a certain group using a strategy for a particular problem 
(e.g., 39% of students in the first u*y group used the strategy of Rereading for Problem 3). 
Therefore, for each analysis, there were six observations (two groups, Uiree problems) for 
every strategy, 

HYPOTHESES 

There were two hypotheses that were tested using the above methodology. First, 
it was hypothesized thai a significant effect would be found for the independent variable 
of number of tries needed to solve a problem; specifically, strategy use for each category 
of su-ategies was hypothesized to increase with the number of tries for a problem. 
Second, it was conjectured that there would be a significant effect resulting from the 
strategies used within each category of strategies; that is, that some strategies fore^ich 
category would prove to be either significantly more or less accessible for the students of 
the sample. Specifically, ha.^d on the research of Lester and Garofalo (1982), 
elementary-age students would not he expected to routinely employ the strategies of 
Analysis of information and conditions, Monitoring progress of local/global [>]ans, and 
Checking for reasonableness. 

KKSIII/iS 

Think-aloiid prolncols ami vidcnIiiptHi uctinns Viuied wiilely across individual 
students in the sample. Examples of proiocols or actions rncmiiilcred during the study, 
which serve to illustrate the use of each sirulegy. art' piusenird in Table 1, 
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Insert Table 1 About Here 



Figure 2 breaks down, by category, student strategy-use for the word problems of the 
study. These bar graphs are intended as a helpful reference for the discussion that follows 
on the ANCOVA results. 



Insert Figure 2 About Here 



Table 2 shows each strategy category's adjusted mean percentages for the 
independent variables. For each category, there were no significant interactions between 



Insert Table 2 About Here 



the covariate and the independent variables, indicating that the use of a particular strategy 
and the number of tries required to solve the problem were not significantly influenced by 
the type of problem being solved. Additionally, none of the analysas produced a 
sigciificani interaction between the independent variables, encouraging investigation of 
the main effects for each category. 

For all categories of strategies, use of a particular strategy was the only significant 
main effect (p<,01 in each case). Thus, the first hypothesis concerning a signilicant main 
effect for number of tries needed to solve a problem was not supported by ihi* data. 'Hie 
main effect of strategies for the different categories resulted in multiple-R^ values of .72 
for Orientation, .67 for Organization, .77 for Execution and .59 for Verification. Based 
on the grand means of Table 2, the Organization strategies were used by the largest 
percentage of students (56.07%), while the Verification strategies were used by the 
smallest percentage ( 1 8. 17%). 
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Tukey's HSD was calculated for each category of .strategies in Table 2 in order to 
determine which strategies differed significantly from each other (p<.05). The strategies 
that resulted from some type of task prompt (Comprehension and Performance of local 
actions) have not been included in th allowing list. For the category of Organization, 
Drawing a diagram was found to he significantly different from both Global planning and 
Local planning; however. Drawing a diagram was utilized less often than the other 
Organization strategies, primarily because it was only applicable for one problem. Under 
the category of Execution, Vocalizing calculations differed significantly from Crossing- 
out discarded calculations. Vocalizing calculations may at first seem to be a natural 
outcome of the think-aloud protocols, but recall that students wen:; only instructed to 
verbalize their thoughts during problem solving, not their visible processes. Finally, for 
the category of Verification, Evaluation of orientation and organization differed 
significantly from both Checking for reasonableness and from Estimation before solution, 
which were both under-utilized by the students of the sample. Thus, in terms of the 
second hypothesis* certain of the strategies did emerge as being significantly more or less 
accessible for students; however, whereas the strategies of Analysis of information and 
conditions. Monitoring progress of local/global plans, and Checking for rea^onableness 
were all hypothesi/x^d to be strategies that would be used mfrequcntly, only Checking for 
reasonableness was used significantly less often. 

Data relating to group membership (first try/more than one try) were also 
analyzed to further clarify the determining factors of strategy-use. Overall, first-try 
success rates for each problem were 82% for Problem 1 , 65% for Problem 2, and 77%^ for 
Pi'oblom 3. Tabic 3 presents adjusted mean percentages for strategy by number of tries 
within eiich category of strategies. Though there was a general trend across all categories 
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toward the percentage of studenls using strategics to increase with ihc number of solution 
attempts, only the Organization strategies did so exclusively. Differences between groups 
were most pronounced for the categories of Orientation and Verification. Interestingly, 
Assessment of familiarity with task and Pen movements were only employed by those 
studenls correctly solving problems using more than one try, though Pen movements 
occurred only once. 

In general, strategy-use for the more-than-one-try group resulted from student 
solutions becoming more thoughtful and careful after the first try. Strategy-use certainly 
was prompted by problem context; however, the use of particular problem-solving 
strategies appeared at times to be more directly affected by studenLs' knowledge of an 
incorrect answer (resulting in a second or third try). 

DISCUSSION 

It should not be assumed that the Figure 1 checklist represents some defmitive 
taxonomy of problem-solving behaviors, or that this same checklist can or should be used 
without modification for students of differing age groups. This checklist resulted from a 
synthesis of sources (Garofalo & Lester, 1985; Lester, Garofalo & Kroll, 1989) which 
focused on studenls and mathematical lasks similar to those in the present study . Two 
general observations can be made concerning the study's sample. 

First, within a category of strategies, particular strategies proved to be more 
natural or accessible for students. For this study, use of Voeali/ing calculations was the 
most common of all of the unprompted Execution strategies; while Evaluation of 
orientation and organization distinguished itself as the most prevalent of the Verification 
strategies. Note that this result suggests neither the most effective nor the most efficient 
problem-solving strategics, since for many students the problems were not solved until 
the second or third try. Additionally, no instruction in problem-solving strategies 
preceded the videotaping. The method of analysis presented in this paper should prove 
beneficial in an experimental setting where an instructional ireatmenl takt'<^ nlacc before 
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ihe problem solving (e.g., Lester, Garofalo & Kroll. 19S9), such thai it is possible to 
compare the strategy-use of those students receiving the treatment with those in a control 
condition. 

Second, an opportunity for self-correction seemed to play a role in determining 
sludeni problem-solving behaviors. Students in this study enhanced their own strategy- 
use merely as a result of being told that their solutions were incorrect and being afforded 
additional opportunities to solve the problems. This was especially true of Assessment of 
familiarity with task within the Orientation category. This result supports the 
constructivist notion that students should be given time to examine their own work 
critically, engaging in what Confrcy (1990) calls the "reflective process;" and that when 
these reflections are captured on videotape they represent a rich X3urcc of information 
about student problem-solving behaviors. 

Overall, application of the strategies checklist to the problem of analyzing 
protocols and actions proved to be sufficient for classifying the problem solving 
behaviors of this sample of sixth grac c students. Data from the checklists were suitable 
for statistical analysis, and results of the analysis provided useful information concerning 
strategy-use across individuals. 
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Figure L Strategies Checklist. 

Student # , IVoblcm # 



Orientation (assessment and understundinK of ttie situution): 

Comprehension 

Analysis of informalion and aimlitions 

Assessment of familiarity with tusk 

Initial and subsequent represcnluiiim 

Assessment of level of difficulty and cliances of success 

Rereading 

Listing data 

Pen movements 

Organization (planning behavior and chuiising actions): 

Identification of goals and subgoals 

(jlohal planning 

Local planning (implementing global plan) 

Drawing a diagram 

Execution (regulation of behavior): 

Performance of local actions 

Monitoring progress of local/global plans 

Vocali/.ing calculations 

Crossing-out discarded calculations 

Subtotaling or tallying 

vShifting gears 

Verification (evaluation of decisions and outcomes): 

Evaluation of orientation and organization 

Evaluation of execution 

Checking for reasonableness 

Estimation before solution 



Adapted from Garofalo and Lester, 1985; Lester, et ah, 1989 
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Tuhle I 

Examples of Protocqis Aettons for the Different Strategy Cateeories 
Orientation 

Comprehension - Reading the problem aloud for the first time. 

Analysis of information and conditions - "Mrs. Jones bought 10 ears of com 

doesn't have anything to do with it/' 

Assessment of familiarity with task - "Not sure I know how to solve this problem," 
Initial and subsequent representation - *The second [shelf] is the same as the last one." 
Assessment of level of difficulty and chances of success - "Pretty easy problem " 
Rereading - Reading the problem aloud for a second, third or fourth time. 
Listing data - Writing out information stated in the problem. 
Pen movements - Using the pen/pencil to attend to words or calculations. 
Organization 

Identification of goals and suhgoals - "You have to find out how many cents one 
orange is worth." 

Global planning - "...how many times a dozen goes into 30." 
Local planning - "First you divide 12 into 30." 

Drawing a diagram - Attempting a pictorial representation of the problem. 
Execution 

Performance of local actions - Carrying out planned mathematical operations. 
Monitoring progress of local/global plans - "Now I have 6 more oranges left" 
Vocalizing calculations - "2 feet plus 3 feet plus 3 feet gives you a total of 6 plus 2 
is 8 feet." 

Crossing-out discarded calculations - Erasing or marking out previous work that is 
no longer relevant. 

Suhtotaling or tallving - Mainiaining a running total on paper during calculations. 
Shifting gears - "Km gonna try and do this over." 
Verification 

Evaluation ol orientation and organi/ation - "1 misunderstood the question." 
Evaluation ol' execution - "(jonna see if I divided right." 
Checking for reasonableness - "Can't be right. Can't have a remainder." 
Estimation before solution - "IT I'm correct it should he about 8 - 19 or 20." 
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Figure 2 . Incidents of Strategy Use by Category and Problem (n=17). p,nhhMn 
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Table 2 

Adjusted Mean Percentages for St rategy and Number of Tries 

Orientation (Grand Mean = 30.08) 
Numhcr of Tries: One More Than One 

24.04 36.13 



Strategy : Comp. Analysis Assess. Represent. Assess. Rereading Listing Pen 

of Into ofFam. ofDiff. Data Move. 



1(H).(K) 3«.(K) 



9.67 



3().(H) 



9.17 



35.83 



13.83 4. 17 



Organization (Grand Mean = 56.67) 

Number of Tries: One More Than One 

52.92 60.42 

Strategy: IDofGoal.s Global Local Use of a 

«& Subgoals Planning Planning Diagram 

47.50 81.50 82.67 15.(M) 



Execution (Grand Mean = 32.29) 
Number of Tries: One More Than One 

29.67 



Strategy: Local Monitor. 

Actions of Prog. 

1(X).(K) 19.67 



34.90 

Vocali/e 
Calc. 

59.67 



Cross-out 
Caic. 

9.67 



Shiftint! 
Gears 

18.17 



Subtotal. 



18.83 



Verification (Grand Mean =18.17) 

Number of Tries: (.>ne More Than One 

10.67 25.67 

Strdlegy: Eval. of Eval. of Check of 

Orient./Org. Execution Reasonable. 



Estimation 
Before Solution 



40.67 



24.(H) 



2.50 



5.50 
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Table 3 

Adjusted Mean Percentages for Strategy by Number of Tries 



Orientation 



Stratepies 



Comp. Analysis Assess. Represenu Assess. Rereading Listing Pen 

of Info ofFam. ofDiff, Data Move. 

No. of 
Tries 
One 
More 

Than One 1(X).(K) 33.33 19.33 39.0() 



1(X).(X) 42.67 



0.(K) 2I.(X) 



7.33 16.()0 5.33 (UK) 
ll.(K) 55.67 22.33 8.33 



Organization 



No. of 
Tries 
One 
More 
Than One 



ID of Goals 
& Subgoals 

45.00 
50.00 



Strategies 



Global 
Planning 

79.67 
83..33 



Local 
Planning 

82.(K) 
83.33 



Use of a 
Diagram 



5.(K) 
25.(X) 



Execution 



No. of 
Tries 
One 
More 
Than One 



Local 
Actions 



1(H).(X) 
KXMX) 



Monitor, 
of Prog. 



22.67 
16.67 



Strategies 

Vocalize 
Calc. 



Cross- out 
Calc. 



64,(K) 
55.33 



2.67 
16.67 



Shifting 
Gears 



3.(H» 
33.33 



Subtotal. 

15.33 
22.-33 



Verification 



Siratceics 



Eval. of Eva;, of Check of 

Orient./Org. Execution Reasonable. 

No. of 
Tries 

One 23.(K) 14.67 5.(K) 

More 

Than One 58.33 3.3.33 ().(X) 



Estimation 
Before Solution 



().(M) 
11. (X) 



